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1.  Introduction 


1.1  Background 

Unmanned  vehicles  (UVs)  are  being  used  more  frequently  in  military  operations,  and  the  types 
of  tasks  they  are  being  used  for  are  evolving  in  complexity.  In  the  future  battlefield.  Soldiers 
may  be  given  multiple  tasks  to  perfonn  concurrently,  such  as  navigating  a  UV  while  conducting 
surveillance,  maintaining  local  security  and  situational  awareness,  and  communicating  with 
fellow  team  members.  To  maximize  human  resources,  it  would  be  ideal  to  designate  a  single 
operator  to  supervise  multiple  UVs  simultaneously.  However,  research  has  shown  that  human 
operators  are  often  unable  to  control  multiple  robots/agents  simultaneously  in  an  effective  and 
efficient  manner  (Chen,  Durlach,  Sloan,  and  Bowens,  2008;  Schurr,  2007).  Additionally,  as  the 
size  of  the  robot  team  increases,  the  human  operators  may  fail  to  maintain  adequate  situational 
awareness  when  their  attention  has  to  constantly  switch  among  the  robots,  and  their  cognitive 
resources  may  be  overwhelmed  by  the  intervention  requests  from  the  robots  (Wang,  Wang,  and 
Lewis,  2008;  Wang,  Lewis,  Velagapudi,  Scerri,  and  Sycara,  2009).  Wang  et  al.  (2009)  reviewed 
a  number  of  studies  on  supervisory  control  of  multiple  ground  robots  for  target  detection  tasks 
and  concluded  that  “the  Fan-out  plateau  lies  somewhere  between  4  and  9+  robots  depending  on 
the  level  of  robot  autonomy  and  environmental  demands”  (p.  143). 

Research  has  shown  that  autonomous  cooperation  between  robots  can  aid  the  performance  of  the 
human  operators  (Wang  et  al.,  2008)  and  enhance  the  overall  human-robot  team  performance 
(Schurr,  2007).  Wang  et  al.,  (2009)  suggested  that  automating  navigation-related  tasks  (e.g., 
path-planning)  is  more  important  than  “efforts  to  improve  automation  for  target  recognition  and 
cueing”  (p.  146)  in  the  context  of  controlling  a  large  team  of  robots.  However,  in  the  foreseeable 
future,  human  operators’  involvement  in  mixed-initiative  (i.e.,  human-robot)  teams  will  always 
be  required,  especially  for  critical  decision  making.  Human  operators’  decision  making  may  be 
influenced  by  “implicit  goals”  about  which  the  robots  are  not  aware  (i.e.,  the  goals  are  not 
programmed  into  the  behaviors  of  the  robots;  Linegang  et  al.,  2006).  In  addition,  the  real-time 
development  on  the  battlefield  may  require  the  human  operator  to  change  the  plan  for  the  robot 
team  and/or  for  the  individual  robots.  Therefore,  effective  communication  between  the  human 
operator  and  the  robots  is  critical  in  ensuring  mission  successes. 

Research  has  been  conducted  on  ways  to  enhance  human-robot  communication  (Stubbs, 
Wettergreen,  and  Nourbakhsh,  2008).  For  example,  researchers  at  Carnegie  Mellon  University 
demonstrated  the  effectiveness  of  a  robot  proxy  to  enhance  shared  understanding  between  the 
human  operator  and  the  robot  in  an  exploration  task  (Stubbs  et  al.,  2008).  The  communication 
mechanism  was  based  on  a  common  ground  collaboration  model  and  was  able  to  improve  the 
human  operator  perfonnance  in  the  following  areas:  more  accurate  plans,  more  efficient  planning 
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(fewer  planning  repetitions),  faster  and  more  efficient  task  performance,  and  better  mental 
modeling  of  the  capabilities  of  the  robot  (Stubbs  et  al.,  2008). 

1.2  Current  Study 

In  the  current  study,  we  investigated  whether  RoboLeader,  a  robotic  surrogate  for  the  human 
operator,  and  an  intelligent  agent  that  could  interpret  the  operator’s  intent  and  issue  detailed 
command  signals  to  a  team  of  robots  of  lower  capabilities,  could  enhance  the  overall  human- 
robot  teaming  perfonnance.  With  the  RoboLeader  capabilities,  dependence  on  operator 
instructions  was  reduced  and  the  level  of  autonomy  in  operation  of  UVs  was  improved  by 
implementing  algorithms  including  real-time  path  planning,  cooperative  control,  and  multi¬ 
objective  decision  of  tactical  strategies.  These  algorithms  were  stacked,  and  the  operator  only 
needed  to  make  high-level  decisions  (Chuyuan,  Qu,  Poliak,  and  Falash,  2008;  Howard,  Qu,  and 
Conrad,  2008;  Qu,  Wang,  and  Hull,  2008;  Qu,  Wang,  and  Plaisted,  2004;  Yang,  Qu,  Wang, 
Conrad,  and  Hull,  2007).  These  algorithms  resided  in  RoboLeader  and  enabled  the  operator  to 
control  a  team  of  robots  through  a  single  user  interface.  RoboLeader  was  able  to  assess  the 
feasibility  of  the  operator’s  plans  by  simulating  their  execution. 

The  effects  of  individual  differences  factors  on  operator  performance  were  also  evaluated.  More 
specifically,  the  effects  of  individual  differences  in  spatial  ability  (SpA)  and  perceived 
attentional  control  (PAC)  on  the  operators’  robotics  control  were  investigated,  as  well  as 
multitasking  perfonnance.  Lathan  and  Tracey  (2002)  demonstrated  that  people  with  higher  SpA 
performed  better  in  a  teleoperation  task  through  a  maze;  they  finished  their  tasks  faster  and  had 
fewer  errors.  Lathan  and  Tracey  suggested  that  military  missions  can  benefit  from  selecting 
personnel  with  higher  SpA  to  operate  robotic  devices.  Our  previous  studies  also  found  SpA  to 
be  a  good  predictor  of  the  operator’s  robotics  performance  (Chen  et  al.,  2008).  Additionally,  we 
examined  the  relationship  between  attentional  control  and  multitasking  performance.  Several 
studies  have  shown  that  there  are  individual  differences  in  multitasking  perfonnance,  and  some 
people  are  less  prone  to  performance  degradation  during  multitasking  conditions  (Rubinstein, 
Meyer,  and  Evans,  2001).  There  is  evidence  that  people  with  better  attentional  control  can 
allocate  their  attention  more  flexibly  and  effectively  (Derryberry  and  Reed,  2002).  This  was 
partially  confinned  by  Chen  and  Joyner  (2009)  with  the  caveat  that  other  studies  showed  that 
those  with  low  PAC  actually  performed  better  with  high  false  alann  target  indictors  compared  to 
higher  PAC  participants,  and  worse  when  the  target  indictors  evinced  a  high  miss  rate  (Chen  and 
Terrence,  2009). 
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2.  Method 


2.1  Participants 

Thirty  individuals  (17  males  and  13  females,  mean  age  of  24-73  years)  from  the  Orlando,  FL 
area  participated  in  the  study.  They  were  compensated  $  1 5/hr  for  their  time. 

2.2  Apparatus 
2.2.1  Simulator 

The  Mixed  Initiative  Experimental  (MIX)  Testbed  was  modified  and  used  as  the  simulator 
(Barber,  Davis,  Nicholson,  Finkelstein,  and  Chen,  2008).  The  MIX  Testbed  is  a  distributed 
simulation  environment  for  investigation  into  how  unmanned  systems  are  used  and  how 
automation  affects  performance.  The  Operator  Control  Unit  (OCU)  of  the  MIX  Testbed  was 
modeled  after  the  Tactical  Control  Unit  developed  under  the  U.S.  Army  Research  Laboratory 
(ARL)  Robotics  Collaborative  Technology  Alliance  (figure  1).  This  platform  includes  a  camera 
payload  and  supports  multiple  levels  of  automation.  Users  can  send  mission  plans  or  teleoperate 
the  platform  with  a  joystick  while  being  provided  a  video  feed  from  the  camera  payload.  Typical 
tasks  include  reconnaissance  and  surveillance. 


Figure  1.  RoboLeader  user  interface. 
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2.2.2  RoboLeader  Algorithm 

2.2.2. 1  General  overview.  The  RoboLeader  utility  consists  of  several  components — interface, 
control  loops,  and  path  generator.  The  interface  pulls  all  necessary  vehicle/environmental  data 
from  the  simulation  environment  and  supplies  the  control  loops  and  path  generator  with  all 
required  information.  The  control  loops  support  a  modular  design,  in  that  additional  capabilities 
can  be  added  to  the  system  with  little  or  no  modification  to  existing  systems.  The  path  generator, 
based  on  the  A-Star  algorithm,  was  implemented  with  concepts  from  vector  mechanics.  Because 
vector  mechanics  was  used,  sets  of  search  criterion  were  developed  that  give  the  path  generator 
unique  behavior  as  compared  to  typical  matrix  search  algorithms.  The  path  generator  can  be 
given  a  start  point  and  an  end  point  to  navigate  towards,  and  the  algorithm  will  “home  in”  on  the 
intended  destination.  The  path  generator  has  the  ability  to  wrap  around  the  destination  until  an 
entry  path  is  found. 

While  the  path  generator  is  responsible  for  calculating  new  routes  for  the  vehicle,  the  finished 
path  solution  is  typically  formed  in  segments  and  stitched  together  by  the  control  loops  in  order 
to  reach  the  final  path  solution.  Depending  on  the  situation,  different  path  formation  behaviors 
must  be  exhibited  by  the  utility.  The  control  loops  are  primarily  responsible  for  assigning 
different  behaviors  to  the  finished  path  solution.  Three  typical  vehicle/environmental  conditions 
may  be  encountered  during  reconnaissance  missions,  and  each  requires  different  path  planning 
schemes;  they  are  blocked/hostile  areas,  high  priority  areas,  and  vehicle  disablement.  The 
different  behaviors  exhibited  for  each  of  the  conditions,  and  operations  required  by  the  utility, 
are  summarized  as  follows: 

•  Hostile/blocked  area:  The  RoboLeader  utility  must  determine  the  vehicles  affected  by  this 
condition  by  analyzing  the  pre-detennined  path  routes  to  see  if  any  vehicle  routes  pass 
through  this  area.  This  task  is  executed  by  the  control  loops.  If  a  vehicle  path  is  found  to 
pass  within  a  hostile/blocked  area,  the  control  loops  detennine  start  and  end  points  that 
correspond  to  navigable  locations  outside  of  the  hostile/blocked  area.  These  points  are  then 
passed  to  the  path  generator,  and  a  new  route  is  calculated  tightly  around  the  area  from  start 
point  to  end  point.  It  is  important  to  tightly  navigate  around  the  hostile  area  and  rejoin  with 
the  existing  path  to  salvage  as  much  of  the  original  reconnaissance  route  as  possible.  This 
newly  formed  path  segment  is  passed  back  to  the  control  loops  for  further  processing.  The 
control  loops  create  the  finished  path  solution  by  merging  the  segment  into  the  original 
vehicle  route.  The  finished  solution  is  then  passed  back  to  the  OCU  for  operator  approval 
or  modification. 

•  High  priority  area:  During  a  reconnaissance  mission,  resources  may  need  to  be  urgently 
diverted  to  a  new  high  priority  location.  Under  this  scenario,  the  control  loops  search 
through  all  of  the  vehicles  involved  in  the  mission  and  detennine  which  ones  must  be 
reallocated  to  the  high  priority  location.  The  control  loop  charged  with  handling  this 
scenario  computes  a  start  point  and  end  point  corresponding  to  the  vehicles’  current 
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positions  and  an  entry  point  into  the  high  priority  location,  respectively.  The  path 
generator  computes  a  path  between  these  points  and  passes  the  result  back  to  the  control 
loops,  which  then  merge  the  new  solution  into  the  portion  of  the  original  path  already 
traversed.  The  control  loop  then  determines  an  exit  point  from  the  high  priority  location 
and  also  identifies  the  location  of  the  originally  intended  destination.  The  path  generator  is 
called  up  again,  and  a  second  path  solution  is  fonned,  which  leads  from  the  high  priority 
area  to  the  original  destination.  The  control  loop  then  forms  the  final  solution. 

•  Disabled  vehicle:  In  the  event  a  vehicle  is  disabled,  a  new  route  must  be  formed  from  the 
vehicle’s  current  location  to  the  original  destination  for  use  by  recovery  forces.  The 
control  loop  for  this  condition  first  identifies  the  vehicle’s  current  location  and  assigns  this 
position  as  the  start  point  for  the  path  generator.  The  destination  of  the  vehicle’s  original 
pre-planned  route  is  then  set  as  the  end  point.  The  path  generator  is  then  called  up  to  form 
the  path.  The  control  loop  then  merges  this  new  path  segment  into  the  portion  of  the 
vehicle’s  path,  which  has  already  been  traversed. 

Figure  2  shows  a  general  flow  of  information  through  the  utility.  A  indicates  vehicle  and  or 
environmental  data  supplied  to  the  path  generator,  and  B  represents  the  calculated  path  produced 
by  the  generator  being  returned  to  the  testbed.  The  main  control  loops  are  responsible  for  calling 
the  path  generator  and  all  other  support  functions,  depending  on  the  previously  addressed 
situations.  In  addition  to  this,  the  control  loops  are  also  responsible  for  collecting  and 
configuring  data  from  the  interface,  and  formatting  the  data  for  use  in  the  path  generator.  The 
next  three  sections — interface,  control  loops,  and  path  generator — provide  greater  detail  into  the 
functional  operation  of  these  components. 


Figure  2.  Data  flow  between  testbed  and  path  generator. 

2.2. 2.2  Interface.  The  algorithms  implemented  in  RoboLeader’s  current  design  require  full 
knowledge  of  the  detected  environmental  events  and  various  instantaneous  vehicle  states.  This 
data  is  passed  from  the  MIX  Testbed  to  the  RoboLeader  algorithms  through  an  interface 
connecting  the  two.  The  interface  thus  forms  the  structure  from  which  the  RoboLeader  utility 
polls  data.  The  interface  is  comprised  of  a  set  of  C++  vectors  that  contain  data  classes  that  store 
the  required  information.  Each  of  the  classes,  when  populated  with  data,  is  loaded  into  its 
respective  vector.  In  other  words,  the  vehicle  vector  contains  the  vehicle  classes,  the  waypoint 
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vector  contains  the  waypoint  classes,  and  the  area  vector  contains  the  area  classes.  Each  class 
stores  information  with  regard  to  one  specific  entity.  For  instance,  a  vehicle  class  contains  state 
data  for  only  one  vehicle.  The  vehicle  vector  then  contains  all  of  the  classes  describing  all 
vehicles  in  the  simulation.  The  three  primary  data  classes  of  vehicle,  waypoint,  and  area,  and  the 
data  they  contain,  are  described  next. 

•  Vehicle  Class.  The  vehicle  class  contains  all  vehicle  state  data,  such  as  vehicle 
identification  number,  pre-planned  route  (contained  in  waypoint  class  and  nested  within 
this  class),  disablement  (indicates  if  the  vehicle  has  been  disabled  or  not),  and  the 
environmental  tag,  which  relates  a  vehicle  to  certain  events  in  the  environment. 

•  Waypoint  Class.  The  waypoint  class  contains  waypoint  data  and  waypoint  visited  status 
for  a  specific  vehicle’s  pre-planned  route. 

•  Area  Class.  The  area  class  contains  geographic  position  information  pertaining  to  regions 
of  interest,  such  as  blocked/hostile  areas,  zones  of  operation,  or  high  priority  areas 
requiring  reconnaissance. 

2.2. 2. 3  Control  loops.  The  main  control  loops  query  the  interface  looking  for  vehicles  tagged 
with  hostile,  high  priority,  or  disabled  designations,  and  call  up  the  path  generator  in  the  proper 
sequences  in  order  to  construct  path  solutions  appropriate  for  the  given  situation.  Figure  3 
illustrates  the  flow  of  control  between  the  path  generator,  the  main  control  loops,  and  the 
interface.  Within  the  interface  block,  the  vehicle  and  area  vectors  are  shown,  which  contain  data 
that  triggers  a  specific  loop  (center  block  of  figure  3)  used  to  formulate  different  path  solutions. 
Three  control  loops  are  used,  and  each  loop  accesses  the  vehicles  environmental  tag.  Depending 
on  the  tag,  one  of  the  three  loops  may  be  called  up  to  generate  a  solution  specific  to  one  of  the 
three  conditions.  If  a  vehicle  is  not  tagged,  all  three  loops  ignore  that  vehicle.  Once  a  tag  is 
identified  by  a  specific  loop,  that  loop  will  compute  the  proper  path  generator  parameters  and 
initiate  the  generator.  Once  the  path  generator  forms  a  solution,  each  control  loop  will  merge  the 
new  solution  into  the  original  path  creating  the  final  desired  path.  This  piecing  together  of  new 
paths  with  old  paths  is  dependant  on  the  desired  behavior. 
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Interface 


Control  Loops 


Figure  3.  Interaction  of  RoboLeader  sub-components. 

2.2. 2.4  Path  generator.  The  base  operation  of  the  path  generator  is  described  in  this  section.  As 
mentioned  before,  the  generator  must  be  given  a  starting  and  ending  point  in  order  to  calculate  a 
path.  Once  start  and  end  points  are  assigned,  the  generator  calculates  a  vector  between  these  two 
points  that  remains  constant  throughout  the  path  calculation  (figure  4).  This  vector  is  called  the 
main  resultant  and  forms  a  guide  for  all  of  the  path  generator’s  decision  making  operations.  Once 
the  main  resultant  is  established,  a  second  set  of  vectors  is  calculated,  which  spans  each 
intersection  that  connects  to  the  start  point  and  the  end  point — i.e.,  a  vector  is  calculated  to  the 
end  point  from  each  intersection  that  connects  with  the  start  point.  The  shortest  vector 
magnitude  distance  is  then  chosen,  the  path  generator  moves  to  that  intersection,  and  the  process 
occurs  repeatedly  until  the  destination  is  reached.  Figure  5  shows  a  depiction  of  this  process. 


7 


Figure  4.  Vector/street  grid. 


Figure  5.  Path  generator  operations. 

Each  step  the  path  generator  takes  through  the  street  grid  is  analyzed  based  on  its  connection 
with  adjacent  intersections,  and  a  decision  is  made  based  on  the  shortest  distance  to  the  end  point 
(figure  4).  Because  vectors  maintain  constant  connection  with  the  end  point,  the  generator  can 
“home  in”  on  its  destination  until  an  entry  path  to  the  end  point  is  found.  The  next  section 
describes  the  add-on  behavior  for  which  the  main  control  loops  augment  the  path  generator. 

The  job  of  the  path  generator  is  straight-forward  in  nature.  The  objective  is  to  efficiently 
determine  a  path  from  one  point  to  another  through  a  street  grid.  As  mentioned  before,  however, 
the  control  loops  provide  additional  behavior  to  the  path  generator  and  are  also  responsible  for 
calling  the  correct  processing  function.  The  illustration  in  figure  6  shows  a  simplified  version  of 


8 


the  processes  the  control  loops  must  execute.  On  the  right-hand  side  of  figure  6,  each  of  the 
three  trigger  conditions  that  call  up  the  appropriate  processing  routine  can  be  seen.  Using  the 
high  priority  condition  as  an  example,  figure  7  depicts  a  simplified  flow  of  control  that  occurs 
within  the  “Call  High  Priority  Processing  Routine”.  In  this  case,  the  path  generator  is  called 
twice — once  to  get  to  the  high  priority  area,  and  then  a  second  time  to  navigate  back  to  the 
intended  destination. 


Figure  6.  Main  control  loop. 
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Figure  7.  High  priority  condition  processing  loop. 

The  modular  nature  of  the  RoboLeader  utility  should  be  evident  at  this  point.  New  features  and 
capabilities  can  be  added  to  the  utility  by  simply  adding  a  new  control  loop  whose  operation  is 
triggered  by  specific  data  reaching  the  interface.  Specific  algorithms  comprising  the  new 
capability  are  then  nested  within  that  control  loop. 

2.2.3  Surveys  and  Tests 

A  demographics  questionnaire  (appendix  A)  was  administered  at  the  beginning  of  the  training 
session.  Since  the  RoboLeader  user  interface  employed  several  colors  to  display  the  plans  for 
the  robots,  and  normal  color  vision  was  required  in  order  to  effectively  interact  with  the  system, 
an  Ishihara  color  vision  test  (with  nine  test  plates)  was  administered  via  PowerPoint1 
presentation.  The  RoboLeader  user  interface  employed  several  colors  to  display  the  plans  for  the 
robots  and  normal  color  vision  was  required  in  order  to  effectively  interact  with  the  system.  A 
questionnaire  on  Attentional  Control  (Derryberry  and  Reed,  2002;  appendix  B)  was  used  to 
evaluate  participants’  perceived  attentional  control.  The  Attentional  Control  survey  consists  of 
2 1  items,  and  measures  attention  focus  and  shifting.  The  scale  has  been  shown  to  have  good 


1  PowerPoint  is  a  registered  trademark  of  Microsoft  Corporation,  Redmond,  WA. 
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internal  reliability  (a  =  .88).  The  Cube  Comparison  Test  (Ekstrom,  French,  and  Harman,  1976) 
and  the  Spatial  Orientation  Test  (Gugerty  and  Brooks,  2004)  were  used  to  assess  participants’ 
spatial  ability.  The  Cube  Comparison  Test  required  participants  to  compare,  in  3  min,  21  pairs 
of  six-sided  cubes  and  determine  if  the  rotated  cubes  were  the  same  or  different.  The  Spatial 
Orientation  Test,  modeled  after  the  cardinal  direction  test  developed  by  Gugerty  and  his 
colleagues  (2004),  is  a  computerized  test  consisting  of  a  brief  training  segment  and  32  test 
questions.  Both  accuracy  and  response  time  were  automatically  captured  by  the  program. 
Participants’  perceived  workload  was  evaluated  using  the  computerized  version  of  the  National 
Aeronautics  and  Space  Administration  (NASA)-Task  Load  Index  (TLX)  questionnaire  (appendix 
C),  which  used  a  pairwise  comparison  weighting  procedure  (Hart  and  Staveland,  1988).  The 
NASA-TLX  is  a  self-reported  questionnaire  of  perceived  demands  in  six  areas:  mental,  physical, 
temporal,  effort  (mental  and  physical),  frustration,  and  performance.  Participants  evaluated  their 
perceived  workload  level  in  these  areas  on  10-point  scales,  as  well  as  completing  pairwise 
comparisons  for  each  subscale. 

2.3  Procedure 

Participants  were  randomly  assigned  to  the  RoboLeader  group  or  the  Baseline  (no  RoboLeader) 
group  before  their  session  started.  After  being  briefed  on  the  purpose  of  the  study  and  signing  an 
informed  consent  form,  participants  completed  the  demographic  questionnaire  and  were 
administered  a  brief  Ishihara  color  vision  test  to  ensure  they  had  normal  color  vision.  After  the 
color  vision  test,  participants  completed  the  Attentional  Control  survey  and  the  two  spatial  ability 
tests.  Participants  then  received  training  and  practice  on  the  tasks  they  would  need  to  conduct. 
Training  was  self-paced  and  was  delivered  by  PowerPoint  slides,  which  showed  the  elements  of 
the  OCU,  steps  for  completing  various  tasks,  several  mini-exercises  for  practicing  the  steps,  and 
exercises  for  performing  the  robotic  control  tasks.  Each  participant  had  to  demonstrate  that  he  or 
she  could  recall  all  the  steps  for  perfonning  the  tasks  without  any  help  by  the  end  of  the  training 
session;  the  training  session  lasted  ~1  h. 

The  experimental  session  also  lasted  about  1  h  and  immediately  followed  the  training  session. 
The  experimental  session  had  two  scenarios,  each  lasting  -30  min,  in  which  participants  used 
their  robotic  assets  to  locate  20  targets  (i.e.,  10  insurgents  carrying  weapons  and  10  improvised 
explosive  devices  [IEDs])  in  the  remote  environment.  There  were  four  robots  available  in  one 
scenario  and  eight  robots  in  the  other  scenario.  The  order  of  scenarios  was  counter-balanced 
across  participants. 

When  each  scenario  started,  the  robots  began  following  pre-planned  routes,  at  which  time  the 
operator’s  task  of  monitoring  the  environment  and  detecting  targets  (insurgents  and  IEDs)  began. 
The  robots  did  not  have  Aided  Target  Recognition  capability;  therefore,  the  participants  had  to 
detect  the  10  insurgents  and  10  IEDs  by  themselves.  For  the  insurgent  targets,  participants  were 
instructed  to  use  their  computer  mouse  to  click  on  the  targets  (i.e.,  to  “lase”  them)  as  soon  as 
they  were  detected.  The  “lased”  insurgents  were  then  automatically  displayed  on  the  map.  For 
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the  IED  targets,  however,  participants  clicked  on  an  IED  button  on  the  interface,  and  then 
marked  the  location  of  the  IEDs  on  the  map.  Additionally,  there  were  friendly  dismounted 
Soldiers  and  civilians  in  the  simulated  environment  to  increase  the  visual  noise  for  the  target 
detection  tasks.  The  participants  were  told  that  their  objective  was  to  finish  reconnoitering  the 
area  using  their  robotic  assets  in  the  least  amount  of  time  possible.  Consequently,  when  re¬ 
planning  a  route,  the  participant  and/or  RoboLeader  had  to  consider  both  the  effectiveness  and 
efficiency  of  the  new  route.  For  example,  situations  in  which  a  robot  completed  its  route  quickly 
but  did  not  cover  much  ground  or  covered  a  lot  of  ground  but  was  slow  to  finish  would  be 
suboptimal  in  comparison  to  re-planning  that  efficiently  (i.e.,  less  time)  covered  a  lot  of  ground. 

In  each  scenario,  there  were  six  events  that  required  revisions  to  a  robot’s  current  plan/route. 
Once  an  event  transpired,  the  Baseline  participants  needed  to  notice  that  the  event  had  occurred, 
and  then  re-route  the  robot  that  was  affected  by  the  event.  For  those  in  the  RoboLeader 
condition,  the  RoboLeader  would  recommend  plan  revisions  to  the  operator,  who  could  either 
accept  the  plans  or  modify  them  as  necessary.  Out  of  these  six  events,  three  were  “bottom-up” 
(i.e.,  unanticipated  obstacles  detected  by  the  robots  that  obstructed  their  navigation)  and  three 
were  “top-down”  (i.e.,  intel  that  the  human  operator  received  from  the  intel  network).  Given  that 
the  events  led  to  obstruction  (e.g.,  vehicles  in  the  path,  hostile  area),  the  RoboLeader  and  the 
participant  needed  to  avoid  re-routing  through  these  areas,  in  addition  to  avoiding  areas  where 
insurgents  or  IEDs  were  already  detected.  Additionally,  in  the  RoboLeader  condition, 
RoboLeader  would  recommend  new  routes  for  robots  that  finished  first  if  it  decided  that  the 
overall  mission  time  could  be  reduced  by  redirecting  those  robots  to  the  unsearched  areas. 

Each  scenario  also  contained  five  situation  awareness  (SA)  queries,  which  were  triggered  based 
on  time  progression  (i.e.,  three  minutes  into  the  scenario).  The  SA  queries  included  questions 
such  as,  “which  areas  have  the  robots  searched?”  (participants  were  instructed  to  mark  the 
searched  areas  on  a  blank  map),  “which  of  your  robots  is  the  closest  to  [Area  of  Interest]?”,  etc. 
The  OCU  screen  was  blank  when  an  SA  query  was  triggered,  and  only  the  SA  query  and  the 
answer  box  were  displayed  on  the  screen.  A  list  of  SA  queries  is  provided  in  appendix  D. 

There  were  two-minute  breaks  between  experimental  sessions.  Participants’  perceived  workload 
(NASA-TLX)  was  also  assessed  after  each  experimental  scenario. 

2.4  Experimental  Design  and  Performance  Measures 

The  study  was  a  2  x  2  mixed  design;  RoboLeader  (with  or  without  RoboLeader  [Baseline])  was 
the  between-subject  variable  and  the  number  of  Robots  used  in  the  scenario  (four  vs.  eight)  was 
the  within-subject  variable.  Perfonnance  measures  included  the  number  of  targets  located  and 
identified,  the  operator’s  situational  awareness  of  the  mission  environment,  and  awareness  of  the 
status  of  the  individual  robots.  A  mixed-design  analysis  of  covariance  (ANCOVA)  with 
RoboLeader  (with  or  without  RoboLeader)  as  the  between-subject  factor  and  number  of  Robots 
(four  vs.  eight)  as  the  within-subject  factor  was  used  to  evaluate  the  operator  performance 
differences  among  the  four  conditions.  Participants’  spatial  ability  (SpA),  which  was  a 
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composite  score  of  the  two  spatial  tests,  and  their  attentional  control  survey  score  were  used  as 
co  variates. 


3.  Results 


3,1  Target  Detection  Performance-Insurgents 

Table  1  lists  several  measures  related  to  target  detection  performance  (insurgents  and  IEDs)  and 
SA  queries,  as  well  as  subjective  workload.  The  analysis  revealed  that  the  robot’s  condition 
significantly  affected  the  number  of  insurgent  targets  detected — F(  \  ,26)  =  21.716,/?  <0.0001. 
Participants  detected  significantly  fewer  insurgents  when  there  were  eight  robots  compared  with 
the  condition  when  four  robots  were  available.  Participants  with  higher  SpA  detected 
significantly  more  insurgents  than  did  those  with  lower  SpA — F(  1 ,26)  =  6.633,/?  <0.05  (figure 
8).  The  effects  of  RoboLeader  and  attentional  control  were  not  statistically  significant. 

Table  1.  Operator  task  performance  and  workload  assessments  (standard  deviations 
are  presented  in  parentheses). 


Measures 

Baseline 

RoboLeader 

4  Robots 

8  Robots 

4  Robots 

8  Robots 

Target  Detection 
(Insurgents) 

6.667a 

(1.345) 

5.000b 

(1.690) 

7.000a 

(2.035) 

5.533b 

(2.200) 

Target  Detection 
(IEDs) 

8.333a 

(1.291) 

7.267b 

(1.668) 

8.400a 

(1.502) 

7.000b 

(2.070) 

SA  Queries 

2.3 1 7a 
(0.858) 

1.333b 

(1.289) 

2.433a 

(1.128) 

1.192b 

(1.012) 

Perceived  Workload 

67.38a 

(16.33) 

71.40a 

(18.10) 

61.27a 

(14.85) 

67.1  la 
(14.69) 

Note:  Statistics  with  the  same  superscript  are  not  significantly  different  from  one  another 
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Low  SpA 


High  SpA 


Baseline  (No  RoboLeader  Baseline  (No  RoboLeader 

RoboLeader)  RoboLeader) 


Figure  8.  Insurgent  detection  performance. 

3.2  Target  Detection  Performance-IEDs 

The  analysis  showed  that  participants  detected  significantly  fewer  IEDs  when  they  had  eight 
robots  compared  with  the  condition  when  four  robots  were  available — F(\  ,26)  =  10.129, 
p  <0.005.  Participants  with  higher  SpA  detected  significantly  more  IEDs  than  did  those  with 
lower  SpA — F(  1,26)  =  10.656,  p  <0.005  (figure  9).  The  effects  of  RoboLeader  and  attentional 
control  were  not  statistically  significant.  There  was  a  non-significant  RoboLeader  x  SpA 
effect — F(  \  ,26)  =  2.545,  p  =  0.12.  However,  as  is  shown  in  figure  9,  participants  with  higher 
SpA  performed  slightly  better  with  RoboLeader  compared  with  the  Baseline  condition;  the  same 
pattern  was  not  observed  for  the  lower  SpA  group. 


Figure  9.  IED  detection  performance. 
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3.3  Situation  Awareness  Queries 


The  analysis  revealed  that  participants’  SA  (figure  10)  was  significantly  lower  when  they  had 
eight  robots  compared  with  the  condition  when  four  robots  were  available — F(  1,26)  =  13.309, 
p  <0.005.  None  of  the  other  factors  were  significant. 


Figure  10.  SA  queries. 

3.4  Perceived  Workload 

The  analysis  showed  that  participants  experienced  significantly  higher  workload  when  there 
were  eight  robots  (M=  69.26)  versus  four  robots  (M=  64.32) — 7^  1,26)  =  4.947,  p  <0.05  (figure 
1 1).  Participants  in  the  RoboLeader  group  assessed  their  workload  slightly  lower  (M=  64.1 1) 
than  did  those  in  the  Baseline  group  (M=  69.38).  However,  the  difference  failed  to  reach 
statistical  significance.  Participants  with  higher  attentional  control  rated  their  workload  as 
significantly  lower  than  did  those  with  lower  attentional  control — F(  1 ,26)  =  7.229,  p  <0.05.  The 
latter  group  also  experienced  significantly  higher  workload  on  five  out  of  the  six  subscales  when 
controlling  eight  robots,  with  the  most  differences  in  Frustration,  Effort  level,  and  Temporal 
demand.  Notably,  females  reported  significantly  higher  workload  (4-robot:  M=  74.97;  8-robot: 
M  =  77.05)  than  did  males  (4-robot:  M=  56.18;  8-robot:  M=  63.29) — F(\  ,28)  =  12.162, 
p  <0.005  (figure  1 1).  The  subscale  data  revealed  that  females  were  significantly  more  frustrated 
than  their  male  counterparts,  regardless  of  the  number  of  robots.  When  there  were  four  robots, 
females  thought  their  performance  was  significantly  worse  than  males;  when  there  were  eight 
robots,  females’  mental  workload  was  significantly  higher. 
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RoboLeader) _ RoboLeader) _ 


Figure  11.  Perceived  workload. 

3.5  Operators’  Interaction  With  the  OCU 

Participants’  interaction  with  the  OCU  (e.g.,  clicks  on  the  graphical  user  interface  [GUI])  was 
further  analyzed.  During  the  experiment,  participants  needed  to  click  on  the  smaller  thumbnails 
(i.e.,  streaming  videos  from  the  robots)  to  enlarge  the  video  image  in  order  to  identify  targets. 
Participants  in  the  Baseline  group  made  significantly  more  clicks  on  the  thumbnails  than  did 
those  in  the  RoboLeader  group — F{  1,25)  =  8.329,/?  <0.01.  They  also  made  more  clicks  when 
they  had  eight  robots  compared  with  the  four-robot  condition — F(\ ,25)  =  132.229 ,/?  <0.001. 
Additionally,  there  was  a  significant  RoboLeader  x  Robots  interaction,  as  well  as  a  significant 
Robots  x  SpA  interaction — F(1 ,25)  =  5.637,/?  <0.05  and  F{  1,25)  =  4.291,/?  <0.05,  respectively 
(figure  12).  The  difference  between  the  four-robot  and  eight-robot  conditions  in  the  Baseline 
group  was  greater  than  that  in  the  RoboLeader  group.  Additionally,  participants  with  higher  SpA 
tended  to  make  more  thumbnail  clicks  in  the  Baseline  condition,  but  not  in  the  RoboLeader 
condition.  In  addition  to  differences  in  the  number  of  thumbnail  clicks,  participants  in  the 
RoboLeader  group  spent  significantly  less  time  completing  their  mission  scenarios  than  did  those 
in  the  Baseline  group — F(  1 ,27)  =  7.1 18,/?  <0.05.  Participants  in  the  RoboLeader  group  spent, 
on  average,  20.68  min/scenario;  on  the  other  hand,  those  in  the  Baseline  group  spent 
23.77  min/scenario. 


16 


Figure  12.  Thumbnail  clicks. 


4.  Discussion 


We  developed  an  intelligent  agent,  RoboLeader,  that  could  assist  human  operators  in  controlling 
a  team  of  robots.  More  specifically,  RoboLeader  could  help  the  operators  with  their  route 
planning  tasks.  Although  we  did  not  find  significant  differences  in  target  detection  performance 
between  the  two  groups,  participants  in  the  RoboLeader  group  completed  their  missions  in 
significantly  less  time  than  did  those  in  the  Baseline  group.  On  average,  RoboLeader  saved  the 
participants  ~3  min  for  missions  lasting  20  or  more  min.  This  finding  was  expected,  given  the 
assistance  the  participants  received  from  RoboLeader  in  the  path-planning  tasks.  On  the  other 
hand,  participants  in  the  RoboLeader  group  exhibited  significant  complacency  behavior 
compared  with  their  Baseline  group  counterparts.  More  specifically,  the  RoboLeader 
participants,  especially  those  with  higher  SpA,  made  significantly  fewer  clicks  on  the  thumbnails 
of  the  streaming  videos  from  the  robots  (to  look  for  targets),  compared  with  the  Baseline  group 
participants.  While  the  number  of  clicks  did  not  directly  translate  into  a  greater  number  of 
targets  detected  (i.e.,  the  correlation  was  not  significant),  this  striking  difference  in  the  way  the 
operators  interacted  with  the  OCU  based  on  the  presence  of  RoboLeader  needs  to  be  further 
investigated. 

We  compared  the  operators’  target  detection  performance  in  the  4-robot  and  8-robot  conditions. 
The  results  showed  that  the  participants  detected  significantly  less  insurgents  when  there  were 
eight  robots  compared  to  the  four-robot  condition,  indicating  less  efficiency  with  more 
resources/assets.  Similarly,  the  participants  detected  significantly  less  IEDs  when  they  had  eight 
robots.  Participants  with  higher  SpA  detected  more  insurgents,  as  well  as  more  IEDs.  These 
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results  are  consistent  with  previous  findings  that  individuals  with  higher  SpA  tend  to  exhibit 
more  effective  scanning  performance  and,  therefore,  are  able  to  detect  more  targets  than  those 
with  lower  SpA  (Chen  et  ah,  2008;  Chen  and  Joyner,  2009;  Chen  and  Terrence,  2008,  2009; 

Chen  and  Clark,  2008).  It  is  likely  that  the  utility  of  RoboLeader  was  not  sufficient  to  overcome 
the  effect  of  SpA.  In  other  words,  the  participants  with  higher  SpA  were  able  to  outperform 
those  with  lower  SpA,  regardless  of  the  RoboLeader  condition.  Additionally,  there  was  a  non¬ 
significant  trend  toward  a  difference  between  higher  and  lower  SpA  participants  in  their  IED 
detection  performance  when  they  had  access  to  RoboLeader.  Those  with  lower  SpA  did  not 
seem  to  benefit  from  RoboLeader  as  much  as  their  higher  SpA  counterparts.  It  is  likely  that  the 
lower  SpA  participants’  scanning  of  the  streaming  videos  on  the  OCU  was  more  disrupted  by 
their  interaction  with  RoboLeader;  on  the  other  hand,  the  higher  SpA  participants’  scanning  was 
more  effective  and  less  affected  by  their  interaction  with  RoboLeader.  Future  research  can 
benefit  from  investigating  the  effects  of  SpA  on  the  scanning  behaviors  of  robotics  operators. 

When  there  were  eight  robots,  participants’  SA  was  significantly  worse  than  when  there  were 
only  four  robots.  On  the  other  hand,  the  SA  of  the  RoboLeader  participants  was  not  significantly 
degraded  compared  with  the  Baseline  group.  In  other  words,  the  “out-of-the-loop”  phenomenon 
associated  with  automation  (Chen  and  Joyner,  2009;  Parasuraman,  Molloy,  and  Singh,  1993; 
Young  and  Stanton,  2007)  was  not  manifested  in  the  RoboLeader  condition. 

Participants  experienced  significantly  higher  workload  when  there  were  eight  robots  compared  to 
the  four-robot  condition,  and  those  with  better  attentional  control  reported  lower  workload  than 
did  those  with  poorer  attentional  control.  Females  also  reported  significantly  higher  workload 
than  did  males.  Those  participants  in  the  RoboLeader  group  rated  their  workload  as  slightly 
lower  than  did  those  in  the  Baseline  group,  although  the  difference  did  not  reach  statistical 
significance. 


5.  Transitions 


RoboLeader  will  be  used  to  support  our  Year  Two  (FY10)  Director’s  Research  Initiative  (DRI) 
work,  in  which  the  capabilities  of  RoboLeader  will  be  expanded  to  deal  more  specifically  with 
research  on  dynamic  re -tasking  requirements  for  persistent  surveillance  of  a  simulated  urban 
environment  based  on  various  battlefield  developments  (e.g.,  individual  robots  need  to  be  re¬ 
tasked  to  search  for  a  high-stake  target).  Furthermore,  the  capability  of  RoboLeader  will  be 
extended  beyond  the  coordination  with  homogeneous  assets  (i.e.,  unmanned  ground  vehicles 
[UGVs])  to  coordination  with  heterogeneous  assets  (i.e.,  unmanned  aerial  vehicles  [UAVs]  and 
UGVs)  when  in  pursuit  of  moving  targets  in  urban  environments.  Additionally,  RoboLeader  will 
be  used  for  studies  of  automation  reliability  and  operator  individual  differences  in  future  Safe 
Operations  for  Unmanned  Reconnaissance  in  Complex  Environments  (SOURCE)  Anny 
Technology  Objective  (ATO)  research. 
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Appendix  A.  Demographic  Questionnaire 


This  appendix  appears  in  its  original  fonn  without  editorial  changes. 
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Participant  # 


Age 


Major 


Date 


Gender 


1.  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _  Other _ 

2.  When  did  you  use  computers  in  your  education?  ( Circle  all  that  apply ) 

Grade  School  Jr.  High  High  School 

Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer?  ( Circle  all  that  apply ) 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

How  often  do  you: 

Use  a  mouse?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  a  joystick?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Use  a  touch  screen?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  icon-based  programs/software? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  programs/software  with  pull-down  menus? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  graphics/drawing  features  in  software  packages? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 
Use  E-mail?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Operate  a  radio  controlled  vehicle  (car,  boat,  or  plane)? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Play  computer/video  games? 

Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

5.  Which  type(s)  of  computer/video  games  do  you  most  often  play  if  you  play  at  least  once  every  few  months? 

6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 

_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 

7.  Are  you  in  your  usual  state  of  health  physically?  YES  NO 
If  NO,  please  briefly  explain: 

8.  How  many  hours  of  sleep  did  you  get  last  night? _ hours 

9.  Do  you  have  normal  color  vision?  YES  NO 

10.  Do  you  have  prior  military  service?  YES  NO  If  Yes,  how  long _ 
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Appendix  B.  Attentional  Control  Survey 


This  appendix  appears  in  its  original  form  without  editorial  changes. 
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For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

It  is  very  hard  for  me  to  concentrate  on  a  difficult  task  when  there  are  noises  around. 

Almost  never,  Sometimes,  Often,  Always 

When  I  need  to  concentrate  and  solve  a  problem,  I  have  trouble  focusing  my  attention. 

Almost  never,  Sometimes,  Often,  Always 

When  I  am  working  hard  on  something,  1  still  get  distracted  by  events  around  me. 

Almost  never,  Sometimes,  Often,  Always 

My  concentration  is  good  even  if  there  is  music  in  the  room  around  me. 

Almost  never,  Sometimes,  Often,  Always 

When  concentrating,  I  can  focus  my  attention  so  that  I  become  unaware  of  what’s  going  on  in  the  room  around 
me.  Almost  never,  Sometimes,  Often,  Always 

When  I  am  reading  or  studying,  I  am  easily  distracted  if  there  are  people  talking  in  the  same  room. 

Almost  never,  Sometimes,  Often,  Always 

When  trying  to  focus  my  attention  on  something,  I  have  difficulty  blocking  out  distracting  thoughts. 

Almost  never,  Sometimes,  Often,  Always 

1  have  a  hard  time  concentrating  when  I’m  excited  about  something. 

Almost  never,  Sometimes,  Often,  Always 

When  concentrating,  I  ignore  feelings  of  hunger  or  thirst.  Almost  never,  Sometimes,  Often,  Always 

I  can  quickly  switch  from  one  task  to  another.  Almost  never,  Sometimes,  Often,  Always 

It  takes  me  a  while  to  get  really  involved  in  a  new  task.  Almost  never,  Sometimes,  Often,  Always 

It  is  difficult  for  me  to  coordinate  my  attention  between  the  listening  and  writing  required  when  taking  notes 
during  lectures.  Almost  never,  Sometimes,  Often,  Always 

I  can  become  interested  in  a  new  topic  very  quickly  when  I  need  to. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  read  or  write  while  I’m  also  talking  on  the  phone. 

Almost  never,  Sometimes,  Often,  Always 

I  have  trouble  carrying  on  two  conversations  at  once.  Almost  never,  Sometimes,  Often,  Always 

I  have  a  hard  time  coming  up  with  new  ideas  quickly.  Almost  never,  Sometimes,  Often,  Always 

After  being  interrupted  or  distracted,  I  can  easily  shift  my  attention  back  to  what  I  was  doing  before. 

Almost  never,  Sometimes,  Often,  Always 

When  a  distracting  thought  comes  to  mind,  it  is  easy  for  me  to  shift  my  attention  away  from  it. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  alternate  between  two  different  tasks.  Almost  never,  Sometimes,  Often,  Always 

It  is  hard  for  me  to  break  from  one  way  of  thinking  about  something  and  look  at  it  from  another  point  of  view. 

Almost  never,  Sometimes,  Often,  Always 
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Appendix  C.  NASA-TLX  Questionnaire 


This  appendix  appears  in  its  original  form  without  editorial  changes. 
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Please  rate  your  overall  impression  of  demands  imposed  on  you  during  the  exercise. 


1.  Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  looking, 
searching,  etc.)?  Was  the  task  easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

2.  Physical  Demand:  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning, 
controlling,  activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or 
laborious? 


LOW  l-l-l-l-l-l-l-l-l-l  HIGH 

123456789  10 

3.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  task  or 
task  elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

4.  Level  of  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 


LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

5.  Level  of  Frustration:  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent  did  you  feel  during  the  task? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 
123456789  10 

6.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

LOW  l-l-l-l-l-l-l-l-l-l  HIGH 

123456789  10 
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Appendix  D.  SA  Queries 
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•  Use  the  provided  map  to  identify  where  your  robots  are  currently  located 

•  Use  the  provided  map  to  identify  which  robot  has  searched  the  highlighted  area 

•  Which  robot  has  encountered  the  most  IEDs? 

•  Which  robot  has  encountered  the  most  insurgents? 

•  What  was  the  name  of  the  last  route  you  edited? 

•  Which  route  was  edited  to  perform  reconnaissance  in  a  High  Priority  Area? 

•  Which  route  has  been  edited  to  avoid  a  Hostile  Area? 

•  Which  robot  is  closest  to  finishing  their  route? 

•  Which  route  has  encountered  an  IED  explosion? 
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List  of  Symbols,  Abbreviations  and  Acronyms 


ANCOVA 

Analysis  of  Covariance 

ARL 

U.S.  Army  Research  Laboratory 

ATO 

Army  Technology  Objective 

DRI 

Director’s  Research  Initiative 

IED 

improvised  explosive  device 

MIX 

Mixed  Initiative  Experimental 

OCU 

operator  control  unit 

PAC 

perceived  attentional  control 

SA 

situation  awareness 

SpA 

spatial  ability 

TLX 

Task  Load  Index 

UV 

unmanned  vehicle 

UAV 

unmanned  aerial  vehicle 

UGV 

unmanned  ground  vehicle 
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NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  HRR 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIM  L 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIM  P 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1197 


ABERDEEN  PROVING  GROUND 

1  DIR  USARL 

RDRL  CIM  G  (BLDG  4600) 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  A  J  MARTIN 
MYER  CENTER  BLDG  2700  RM  2D3 1 1 
FORT  MONMOUTH  NJ  07703-5601 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 
320  MANSCEN  LOOP  STE  115 
FORT  LEONARD  WOOD  MO  65473 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI  T  DAVIS 
BLDG  5400  RM  C242 
REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DG  R  SPINE 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AJ  J  HANSBERGER 
JFCOM  FE 

115  LAKEVIEW  PKWY  STE  B 
SUFFOLK  VA  23435 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
NATICK  SOLDIER  CTR 
AMSRDNSCWSE  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  11  MILERD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE  RM  348 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  283 10-5000 
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NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  DW  E  REDDEN 
BLDG  4  CL  60 

FORT  BENNING  GA  31905-5400 

1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

5  DIR  USARL 
RDRL  CIM  G 
S  FOPPIANO 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM  B 
J  LOCKETT 
RDRL  HRS  D 
B  AMREIN 
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